Removal of Artifacts in Flash Images

ABSTRACT

The present invention relates to a method, a computer-readable medium, a computer program and apparatuses for removal of artifacts in flash images. Image data of a reference image captured using a first level of flash energy and image data of a main image captured using a second level of flash energy that is higher than said first level of flash energy is received. Image data of a third image is then determined based on said image data of said reference image and said image data of said main image, wherein said third image is a representation of said main image with removed artifacts.

FIELD OF THE INVENTION

This invention relates to a method, a computer-readable medium, acomputer program and apparatuses for removal of artifacts in flashimages.

BACKGROUND OF THE INVENTION

A well-known and particularly annoying artifact in flash images is theso-called “red-eye effect”, where people's eyes appear red instead oftheir natural color. Therein, the red color stems from a reflection ofthe flash light on the blood-rich retina. The effect is particularlypronounced when the flash light is arranged near the optical axis of thecamera lens, which is particularly the case when deploying small-sizedcameras, such as for instance compact cameras or cameras integrated intomobile appliances (e.g. mobile phones).

The red-eye effect may be combated in a plurality of ways.

A particularly simple way is to move the flash light away from theoptical axis of the camera lens, so that the camera lens does no longerreceive the direct reflection of the flash light from the retina.However, this approach is in general not feasible for compact cameras orcameras that are integrated into mobile appliances due to sizeconstraints.

According to a further approach, described in U.S. Pat. No. 4,285,588, apre-flash is used that causes the pupil to close before the actual imageis captured. The time between the pre-flash and the actual capturing ofthe image has to be chosen large enough to allow the pupil to close. InU.S. Pat. No. 4,285,588, a time delay of around 600 ms is preferred.This however increases the latency of image capture, and furthermore isnot suited to entirely remove the red-eye effect, since the pre-flashwill not cause the pupil to close completely.

Removal of the red-eye effect may furthermore be accomplished by analgorithm processing a captured image and attempting to identify andremove red eyes contained therein, for instance based on geometricalconstraints prescribed by the form of the eye. However, both the rate offalse and missed detections of red eyes is generally quite high.

U.S. patent application publication no. 2006/0008171 describes a furtherapproach for red-eye removal, which relies on taking an image pair,comprising an image taken without flash and a subsequent image takenwith flash. Therein, both images are taken in short succession, forinstance within 1/30 of a second. The difference in red chrominancebetween the no-flash image and the flash image is determined, and, basedon a threshold value, it is determined which regions may form potentialred eyes. Subsequently, the red-eye regions are removed. However, theperformance of this approach critically depends on the choice of thethreshold value. Furthermore, performance further significantly degradesif the no-flash image is too dark.

SUMMARY

It is thus, inter alia, an object of the present invention to provide amethod, a computer-readable medium, a computer program and an apparatusfor reducing artifacts in flash images.

According to a first aspect of the present invention, a method isdescribed, comprising receiving at least image data of a reference imagecaptured using a first level of flash energy and image data of a mainimage captured using a second level of flash energy that is higher thanthe first level of flash energy; and determining image data of a thirdimage at least based on the image data of the reference image and theimage data of the main image, wherein the third image is arepresentation of the main image with removed artifacts.

According to a second aspect of the present invention, acomputer-readable medium having a computer program stored thereon isdescribed, the computer program comprising instructions operable tocause a processor to receive at least image data of a reference imagecaptured using a first level of flash energy and image data of a mainimage captured using a second level of flash energy that is higher thanthe first level of flash energy; and instructions operable to cause aprocessor to determine image data of a third image at least based on theimage data of the reference image and the image data of the main image,wherein the third image is a representation of the main image withremoved artifacts.

According to a third aspect of the present invention, a computer programis described, comprising instructions operable to cause a processor toreceive at least image data of a reference image captured using a firstlevel of flash energy and image data of a main image captured using asecond level of flash energy that is higher than the first level offlash energy; and instructions operable to cause a processor todetermine image data of a third image at least based on image data ofthe reference image and image data of the main image, wherein the thirdimage is a representation of the main image with removed artifacts.

According to a fourth aspect of the present invention, an apparatus isdescribed, comprising a processor configured to receive at least imagedata of a reference image captured using a first level of flash energyand image data of a main image captured using a second level of flashenergy that is higher than the first level of flash energy; and todetermine image data of a third image at least based on image data ofthe reference image and image data of the main image, wherein the thirdimage is a representation of the main image with removed artifacts.

According to a fifth aspect of the present invention, an apparatus isdescribed, comprising means for receiving at least image data of areference image captured using a first level of flash energy and imagedata of a main image captured using a second level of flash energy thatis higher than the first level of flash energy; and means fordetermining image data of a third image at least based on image data ofthe reference image and image data of the main image, wherein the thirdimage is a representation of the main image with removed artifacts.

According to the present invention, at least image data of the referenceimage and the main image is received and used to determine image data ofa third image, wherein the third image is a representation of the mainimage with removed artifacts. Equally well, image data of more than twoimages may be received and used to determine the image data of the thirdimage. The removal of the artifacts may be understood as a substantialor complete removal of the artifacts. The reference image has beencaptured with a lower level of flash energy than the main image. If morethan two images are captured, the additional images (i.e. the third,fourth, etc. image) may be captured with or without flash. For instance,a viewfinder image may serve as an additional image.

The reference image may either have been captured before the main image,at least partially together with the main image (for instance with twoimage sensors or with an image sensor that is capable of capturing twoimages at a time), or may have been captured after the main image.Capturing the low-flash-level reference image after the high-flash-levelmain image may for instance be advantageous since the facial expressionof image targets may be distracted by the low-level pre-flash. If theartifacts contained in the main image comprise the red-eye effect, thelower level of flash energy may for instance be chosen low enough sothat no or only a negligible red-eye effect occurs in the referenceimage. However, using a flash when capturing the reference image ensuresthat determining the image data of the third image, when exemplarilybeing based on differences between image data of the reference image andthe main image, yields adequate results, since it is avoided that thereference image is too dark and thus aggravates a comparison of theimage data of the reference image and the main image. By using flashalso when capturing the reference image, the quality of detection ofpotential artifact regions in the captured images is increased, so thatartifact removal is rendered more robust.

The reference image and the main image may be captured under usage ofdifferent flashlights, i.e. a first type of flashlight for the captureof the reference image and a second type of flashlight for the captureof the main image. The types of flashlights used may for instancecomprise, but not be limited to, a Xenon-flash, an LED-flash, or anindicator light.

The determining of the image data of the third image is performed by aprocessor which receives the image data of the reference image and themain image. Therein, means for capturing the reference image and themain image may for instance be integrated in the same apparatus in whichalso the processor is comprised, or in a separate apparatus. In thelatter case, the processor then may for instance be furnished with aninterface for receiving the image data of the reference image and andthe main image and embodied as a module that can be integrated in anapparatus with a camera unit for capturing the reference image and themain image. The processor may read program code from a computer-readablemedium, such as for instance a fixedly installed or removable memoryunit, wherein the program code comprises instructions operable to causethe processor to receive the image data of the reference image and themain image and to determine the image data of the third image.

The artifacts may for instance comprise the red-eye effect, caused byreflection of flash light at the blood-rich retina of a person's eye.The artifacts may for instance also comprise the effect that flash lightis reflected at the tapetum lucidum encountered in a large group ofanimals (comprising for instance cats and dogs), which causes theseanimals' eyes to unnaturally shine in different colors in the flashimage. Moreover, the artifacts are understood to comprise any effectthat is particularly caused by the use of flash light and changes thenatural appearance or deteriorates the quality of the main image.

The determining of the image data of the third image is based on imagedata of the reference image and the main image. In this way, thedetermining does not only have to rely on image data of the main image,but also may consider changes in the image data between both images.

Therein, the image data may be analog or digital image data. The imagedata may for instance be raw image data as obtained from a camera unit,or may already have been transformed according to a specific color spacemodel, such as for instance the YUV or the RGB model, or already havebeen transformed into a specific image format. The image data may alsorepresent only one or more components of a color space model, such asfor instance the Y, U and V components of the YUV model or the R, G andB components of the RGB model.

According to a first exemplary embodiment of the present invention, atemporal distance between the capture of the reference image and themain image is less than 100 ms. Thus either the main image is capturedless than 100 ms after the reference image, or the reference image iscaptured less than 100 ms after the main image. For instance, in theformer case, since the low-level flash used in the capturing of thereference image is not intended to condition the pupil, the dynamicproperties of the pupil do not have to be considered when defining thistime delay. However, it may be advantageous to define the time delay ina way that the major movement of targets between the capturing of theimages is not possible. A suited value for this time delay may forinstance be 30 ms or even less. According to this first exemplaryembodiment, thus the latency of image capturing can be significantlyreduced while still allowing for artifact removal. The temporal distancebetween the capture of the reference image and the main image mayequally well be larger than 100 ms. For instance, if motion compensationis applied to at least one of the reference image and the main image, itmay be possible to allow much larger temporal distances between thecapture of the reference image and the main image, since the compensatedmotion in the images allows to properly compare the images as a basisfor artifact removal.

According to a second exemplary embodiment of the present invention, thefirst level of flash energy is less than 10 percent of the second levelof flash energy. The first level of flash energy may for instance bedefined small enough so that specific artifacts, such as for instancethe red-eye effect, do not or only to a small degree occur in thereference image, but still large enough to ensure that the referenceimage is not too dark and thus suited for a comparison with the mainimage. The level of flash energy and exposure time for the referenceimage may for instance be determined based on the statistics of aviewfinder image. A suited value for the first level of flash energy mayfor instance be 5 to 10 percent of the second level of flash energy, butmay equally well be significantly smaller. The reference image and themain image may be captured under usage of the same flashlight, and onlydifferent levels of flash energy may be applied. Equally well, differenttypes of flashlights may be used for the capture of the reference imageand the main image, respectively. The first and/or second level of flashenergy may then for instance be determined or bounded by the deployedtype of flashlight. For instance, for the capture of the referenceimage, a flashlight producing a smaller level of flash energy than aflashlight used for the capture of the main image may be used.

According to a third exemplary embodiment of the present invention, thereference image has at least one of a lower quality, a lower samplingrate and a lower resolution than the main image. The lower quality,lower sampling rate or lower resolution may be advantageous since thereference image may then require less memory. The lower quality, lowersampling rate or lower resolution may be achieved during capturing ofthe reference image, or may be achieved after image capture bytransformation of the original image obtained from a camera unit.

According to a fourth exemplary embodiment of the present invention, thedetermining of the image data of the third image comprises detectingartifacts in the main image under consideration of differences betweenthe image data of the reference image and the image data of the mainimage; and correcting the detected artifacts to obtain the image data ofthe third image.

The detecting of the artifacts according to the fourth embodiment of thepresent invention may comprise identifying potential artifacts in themain image based on the image data of the main image; determining falseidentifications among the potential artifacts under consideration of thedifferences between the image data of the reference image and the imagedata of the main image; and excluding the false identifications from thepotential artifacts, wherein the correcting is only performed fornon-excluded potential artifacts. Therein, the identification ofpotential artifacts may for instance be performed by a patternrecognition algorithm, for instance by searching for face-shaped oreye-shaped patterns in the main image.

Therein, the reference image may be processed before the detecting ofthe artifacts in the main image. The processing of the reference imagemay for instance comprise equalization of the reference image so thate.g. motion, different image sizes, different exposures or otherdifferences between the reference image and the main image arecompensated.

Alternatively, the detecting of the artifacts according to the fourthembodiment of the present invention may comprise identifying potentialartifacts in the main image based on the differences between the imagedata of the reference image and the image data of the main image;determining false identifications among the potential artifacts underconsideration of the image data of the reference image or the mainimage; and excluding the false identifications from the potentialartifacts, wherein the correcting is only performed for non-excludedpotential artifacts. Performing the identification of potentialartifacts based on both the reference image and the main image maycontribute to reduce the complexity of the determining of falseidentifications among the potential artifacts.

The correcting of the detected artifacts in the fourth exemplaryembodiment of the present invention may at least partially be based onthe image data of the reference image. For instance, image data of themain image may be replaced by image data of the reference image toobtain the image data of the third image.

According to a fifth exemplary embodiment of the present invention, thedetermining of the image data of the third image comprises performing afactor analysis of a set of data formed from the image data of thereference image and the image data of the main image, and applying atransformation obtained from the factor analysis to the set of data toobtain image data of a fourth image, wherein the image data of the thirdimage is determined at least based on the image data of the fourthimage. Deploying a factor analysis allows to blindly suppress or reverseglobal differences between the reference image and the main image andthus avoids pattern recognition steps. It should be noted that thedetermining of the image data of the third image according to the fifthembodiment of the present invention is also applicable in cases wherethe reference image has been captured without flash, i.e. in cases whereimage data of a first non-flash image and image data of a second flashpicture are received. The image data of the third image may then bedetermined based on the image data of the first non-flash image and theimage data of the second flash image. Equally well, the determining ofthe image data of the third image according to the fifth embodiment ofthe present invention is applicable when more than two images arecaptured and then serve as a basis for the determination of the thirdimage. Therein, said reference image and said main images may becaptured with flash, and further images may be captured with or withoutflash. An example for the latter case is the use of a viewfinder imageas a further image.

The factor analysis may for instance be a principal component analysisof the set of data, the principal component analysis determining acommon part and a different part with respect to the image data of thereference image and the image data of the main image. Therein, thecommon part expresses the greatest variability of the joint signal, andthe different part expresses the corresponding differences.

The transformation may be determined based on a transformation matrixobtained from the principal component analysis and a modifiedtransformation matrix determined to suppress or reverse the differentpart. By applying the transformation matrix (in transposed form) to theset of data, the set of data is thus transformed into a new coordinatesystem, and by applying the modified transformation matrix, the set ofdata is transformed back from the new coordinate system, however in amodified way that suppresses or reverses the different part and thus thedifferences between the image data of the reference image and the imagedata of the main image.

In the fifth embodiment of the present invention, a fuzzy likelihood mapmay be determined based on information from the reference image and themain image, wherein the fuzzy likelihood map indicates whether parts ofthe main image contain an artifact or not, and the image data of thereference image and the image data of the main image may be weightedwith the fuzzy likelihood map before the factor analysis is performed.In the fuzzy likelihood map, there may, for instance for each pixel,exist a value in the range [0,1], whereas the magnitude of the valueexpresses if the pixel is likely to be in an artifact region (e.g. ared-eye region). By weighting the image data of the reference image andthe main image with the fuzzy likelihood map, the image datacorresponding to artifacts is emphasized with respect to the image datanot corresponding to artifacts, thus giving image data corresponding toartifacts more weight in subsequent processing. Applying a fuzzylikelihood may make the use of thresholds unnecessary and producesspatially smooth results, unlike methods where thresholds are appliedand lead to visible edges between areas that are processed to removeimage artifacts and non-processed areas.

In the fifth embodiment of the present invention, the image data of thethird image may at least be based on the image data of the referenceimage, the image data of the main image and the image data of the fourthimage under consideration of the fuzzy likelihood map. Furthermore, aninfluence of the fuzzy likelihood map may be removed from the image dataof the third image.

In the fifth embodiment of the present invention, further an intensityof portions of the third image may be reduced. For instance, if theartifacts comprise the red-eye effect, the processing according to thefifth embodiment of the present invention may decrease color saturationof image data related to pupils. For darkening these pupils, theintensity may be reduced.

In the fifth embodiment of the present invention, the image data mayrepresent components of a color space model, the determining of theimage data of the third image may be performed for at least one of thecomponents of the color space model, and the determining of the imagedata of the third image may be performed for the components separately.

In the fifth embodiment of the present invention, the color space modelmay for instance be the YUV model, and the determining of the image dataof the third image may be performed for the U and V component only. Asan Y component for the third image, then for instance the Y component ofthe main image may be used.

In the fifth embodiment of the present invention, the color space modelmay for instance be the YUV model, and the determining of the image dataof the third image may be performed for the U, V and Y component.

These and other aspects of the invention will be apparent from andelucidated with reference to the embodiments described hereinafter.

BRIEF DESCRIPTION OF THE FIGURES

In the figures show:

FIG. 1 a: a schematic block diagram of an exemplary embodiment of anapparatus according to the present invention;

FIG. 1 b: a schematic block diagram of a further exemplary embodiment ofan apparatus according to the present invention;

FIG. 2: a flowchart of an exemplary embodiment of a method according tothe present invention;

FIG. 3: a schematic illustration of the timing of flash and imagecapture according to an exemplary embodiment of the present invention;

FIG. 4: a flowchart of an exemplary embodiment of step 205 of theflowchart of FIG. 2; and

FIG. 5: a flowchart of a further exemplary embodiment of step 205 of theflowchart of FIG. 2.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 a depicts a schematic block diagram of an exemplary embodiment ofan apparatus 1 a according to the present invention. Apparatus lacomprises a camera unit 16 for capturing images using different levelsof flash energy.

To this end, camera unit 16 comprises an image sensor, such as forinstance a Charge Coupled Device (CCD) or a Complementary Metal OxideSemiconductor (CMOS) image sensor, which is configured to capture imagesprojected onto its surface via according camera optics. Images capturedby image sensor 160 are at least temporarily stored in frame buffer 161,which may be configured to store more than one images at a time. Imagesensor 160 may furthermore be equipped with an analog to digitalconverter for transforming signals representing a captured image intodigital data to be stored in frame buffer 161. It should be noted thatframe buffer 161 may equally well form a functional block of centralprocessor 10, or a functional block of camera unit 16.

Camera unit 16 further comprises a flash unit 162, which is configuredto generate flash light (e.g. flash pulses) with different levels offlash energy, for instance to allow capturing of a reference image witha low level of flash energy, and a main image with a high level of flashenergy. The flash unit may for instance be powered by a flash capacitor.

Furthermore, camera unit 16 is equipped with a shutter unit 163, whichcontrols the opening of a shutter controlling projection of images ontothe surface of image sensor 160. The shutter may for instance be arolling shutter or a global shutter, to name but a few possibilities.Furthermore, the shutter may be implemented mechanically orelectronically.

It is understood by those skilled in the art that camera unit 16 maycomprise further functional units such as for instance an auto focusunit for controlling an auto focus operation of camera unit 16, orsimilar units.

Apparatus la further comprises a central processor 10 for controllingthe overall operation of apparatus 1 a. In particular, central processor10 is configured to control image sensor 160, frame buffer 161, flashunit 162 and shutter unit 163 thus to allow capturing of subsequentimages with different levels of flash energy, for instance a referenceimage with a low level of flash energy, and a main image with a highlevel of flash energy.

Apparatus la further comprises display 11, a user interface 15 and animage memory 13 for storing captured and processed images. Image memory13 may be embodied as fixedly built-in or removable memory, such as forinstance a memory stick or card. Display 11, user interface 15 and imagememory 13 are all controlled by central processor 10.

Central processor 10 may run program code stored in processor memory 12,which may furthermore serve as a data memory of central processor 10,and which may for instance be embodied as Random Access Memory (RAM),Read-Only-Memory (ROM), to name but a few possibilities. Processormemory 12 may equally well be embodied as a memory that is removablefrom apparatus 1 a. The program code stored in processor memory 12 mayfor instance define the way how central processor 10 controls the unitsof apparatus 1 a, and in particular may define how subsequently capturedimages of different flash levels are processed to remove image artifactscontained therein.

Apparatus 1 a may for instance represent a digital camera, where display11 then may function as a viewfinder and as a means for displayingcaptured images, and user interface 15 may comprise interaction elementssuch as a camera trigger, control elements for zooming and controlelements for operating a menu structure. Therein, display 11 may also atleast partially function as user interface, for instance by displaying amenu structure.

Equally well, apparatus la may represent an electronic device that isadditionally furnished with functionality to capture subsequent imageswith different levels of flash energy.

For instance, apparatus la may represent a mobile appliance such as amobile phone, a personal digital assistant or a laptop computer.Therein, central processor 10 may then for instance be the standardprocessor for controlling the functioning of the mobile appliance,display 11 may be its standard display, and user interface 15 itsstandard user interface, such as for instance a keyboard or keypad.Similarly, memories 12 and 13 may be standard components alreadycontained in the mobile appliance. In order to furnish the mobileappliance with the functionality to capture images, camera unit 16 maybe added to the mobile appliance, and the program code in processormemory 12 may be accordingly altered to enable processor 10 to controlcamera unit 16 to capture subsequent images with different levels offlash energy, and to remove image artifacts contained therein.

Moreover, FIG. 1 a illustrates that apparatus 1 a may further comprise adedicated artifact removal processor 14, which is however optional andthus depicted in dashed lines. Since image artifact removal may be aconsiderably complex computation task, it may be advantageous to haveimage artifact removal performed by a dedicated processor architecture,such as for instance an Application Specific Integrated Circuit (ASIC)or a Field Programmable Gate Array (FPGA). The dedicated artifactremoval processor 14 then would receive image data of images capturedwith different levels of flash energy from central processor 10 andwould remove image artifacts contained therein, thus to obtain imagedata of an image without image artifacts.

FIG. 1 b depicts a schematic block diagram of a further exemplaryembodiment of an apparatus 1 b according to the present invention. Incontrast to apparatus 1 a of FIG. 1 a, in apparatus 1 b of FIG. 1 bimage processing is partially or completely outsourced from centralprocessor 10′ to image processor 17. Image processor 17 (which may alsobe denoted as imaging and video engine) may for instance be embodied viahardwired and/or programmable functional blocks for image processingtasks, inter alia comprising image processing to remove image artifactsaccording to the present invention. To this end, image processor 17 maycomprise internal or external memory for storing a computer program withinstructions operable to perform artifact removal according to thepresent invention. Similar to the apparatus 1 a of FIG. 1 a, also in theapparatus 1 b of FIG. 1 b, a dedicated artifact removal processor 14,for instance in the form of a module, may be foreseen to furtherpartially of completely outsource image artifact removal from imageprocessor 17. Furthermore, image processor 17 may directly interfacewith image memory 13.

Apparatus 1 b may for instance represent a digital camera, wherein theunits of apparatus 1 b then function as already explained with referenceto apparatus 1 a of FIG. 1 a above. Equally well, apparatus 1 b mayrepresent an electronic device that is additionally furnished withfunctionality to capture subsequent images with different levels offlash energy; for instance, apparatus 1 b may represent a mobileappliance such as a mobile phone, a personal digital assistant or alaptop computer.

FIG. 2 depicts a flowchart 200 of an exemplary embodiment of a methodaccording to the present invention. The steps of this flowchart may forinstance be performed by central processor 10 (and/or optional dedicatedartifact removal processor 14) of apparatus la (see FIG. 1 a), or bycentral processor 10′ and image processor 17 (and/or optional dedicatedartifact removal processor 14) of apparatus 1 b (see FIG. 1 b). Therein,it is exemplarily assumed that an image of a person is to be capturedusing flash light, so that the artifact that is to be removed from thecaptured images is the red-eye effect, caused by reflection of the flashlight at the blood-rich retina of the person's eye. It is neverthelessunderstood by a person skilled in the art that the present invention isequally well applicable to the removal of other types of artifacts inflash images, such as for instance reflection of the flash light at thetapetum lucidum that is encountered in a large group of animals(comprising for instance cats and dogs) and causes these animals eyes tounnaturally shine in different colors in the flash image. Furthermore,it is exemplarily assumed that the reference image with a low level offlash energy is captured before the main image with a high level offlash energy. Alternatively, the reference image with the low level offlash energy may also be captured after the main image with the highlevel of flash energy.

In a first step 201 of flowchart 200, it is checked if a camera triggerhas been activated by a user. If this is not the case, the flowchartreturns to step 201 in an endless loop to receive any camera triggeractivation.

If it is determined in step 201 that the camera trigger has beenactivated, camera unit 16 is controlled to capture a reference imagewith a low level of flash energy in a step 202. This reference image mayfor instance be taken with only 5-10 percent (or even less) of the flashenergy that was used for the capture of the main image, and with a shortexposure time, for instance 10-50 ms. Therein, the level of flash energymay advantageously be chosen in a way that no red-eye effect or only aminimal red-eye effect is caused, but that the image is still not toodark to allow proper comparison with a main image that will subsequentlybe taken with a higher level of flash energy. The level of flash energyand exposure time for the reference image may for instance be determinedbased on the statistics of the viewfinder image data. It should be notedthat, in case that the viewfinder image is bright enough, it may also bepossible to use the viewfinder image as a reference image and todispense with the capturing of the reference image at all. Similarly, insome cases it may be beneficial to capture the reference image with zeroflash intensity. Furthermore, the reference image may be captured forinstance with a lower quality, a lower sampling rate and a lowerresolution compared to the main image. This may contribute to savingmemory required to temporarily store the reference image in frame buffer161, since the reference image has to be stored in addition to the mainimage.

In a step 203, camera unit 16 is controlled to capture a main image witha high level of flash energy, for instance with 100 percent of anavailable flash capacitor energy. The main image may be captured withfull resolution. The level of flash energy and the exposure time may forinstance be determined based on the statistics of the viewfinder imageand/or the statistics of the reference image. The main image is actuallythe only desired picture. However, due to the use of flash light with ahigh level of flash energy required to adequately lighten a person thatis to be photographed, also the occurrence of the red-eye effect isinevitable. However, based on information from both the reference imageand the main image, the red-eye effect can be removed from the mainimage, thus obtaining a representation of the main image with removedred-eye effect. This removal of the red-eye effect is simplified whenthe time delay between the capturing of the reference image and the mainimage is kept short to avoid major motion and/or content changes. It maythus be advantageous to capture the main image as soon as possible afterthe capturing of the reference image, for instance not more than 100 ms,in exceptional cases also not more than 200 ms after the capturing ofthe reference image. For instance, larger delays between the capture ofthe two images may be possible if motion compensation is applied to oneor both of the two images. It should be noted that the main image andthe reference picture may furthermore be captured with differentexposure times, for instance a shorter exposure time may be used for themain image. Furthermore, it may be advantageous to use the flash in thecapture of the reference image as late as possible and in the capture ofthe main image as early as possible. The positioning of the flashinterval within the image capture interval may also be either fixed orvariable.

In a step 204, image data of the captured reference image and thecaptured main image is received. Therein, said image data may forinstance be one component (e.g. raw data or Y, U, V, R, G, B) or morecomponents of a color space model, such as for instance the YUV model orthe RGB model. It is readily understood by those skilled in the art thatthe image data of the reference image may equally well be receiveddirectly after the capture of the reference image in step 202 and beforethe capture of the main image in step 203. In step 204, then only theimage data of the main image would be received. It then may be possibleto dispense with a frame buffer that is capable of storing image data oftwo images at a time.

In a step 205, image data of a main image with removed red-eye effect isdetermined. Specific embodiments of this step will be discussed in moredetail with respect to the flowcharts 400 of FIGS. 4 and 500 of FIG. 5below. This determining is based on image data of both the reference andthe main image. Therein, it is, inter alia, exploited that the referenceimage captured with a low level of flash energy most likely does notcomprise the red-eye effect, and is taken only shortly before (oralternatively after) the main image, so that a comparison of both imagesto detect changes therein is possible without suffering too much fromartifacts caused by motion and/or scene change.

Finally, in a step 206, the image data of the main image with removedred-eye effect is stored, for instance in image memory 13 (see FIG. 1a/1 b). Optionally, said main image with removed red-eye effect may alsobe displayed on display 11. It may also be possible that the imagewithout removed red-eye-effect is shown on display 11 due to processinglatency caused by the red-eye-effect removal procedure. The flowchartthen returns to step 201.

FIG. 3 schematically illustrates the timing of flash and image captureaccording to an exemplary embodiment of the present invention. The uppergraph 30 illustrates the deployment of a flash light during imagecapture, and the lower graph 31 illustrates the periods of imagecapture. It is readily seen that, in this exemplary embodiment, wherethe reference image is exemplarily captured before the main image, thecapturing 310 of the reference image is performed with a short exposuretime of 20 ms, and with a low energy flash pulse 300, whereas thecapturing 311 of the main image is performed with a longer exposure timeof 50 ms, and with a high energy flash pulse 301. Furthermore, the delaybetween the capturing of the reference image 310 and the capturing ofthe main image 311 amounts to 5 ms only. As already stated above, thepositioning of the intervals where the flashes 300 and 301 are activatedwithin the capturing intervals 310 and 311 are not mandatory. It may forinstance be advantageous to activate the low energy flash pulse 300 atthe end of the capturing 310 of the reference image. Similarly, thecapturing of the reference image 310 and the main image 311 may beperformed with different exposure times. Furthermore, the amplitudes ofthe flashes 300 and 301 may be different.

FIG. 4 presents a flowchart 400 of an exemplary embodiment of step 205of the flowchart of FIG. 2, i.e. an exemplary way how to determine imagedata of the main image with removed red-eye effect.

In a first step 401, the reference image is equalized, for instance byperforming motion compensation, color adjustment, exposure adjustment,scaling or further techniques to make the reference image comparable tothe main image. Step 401 is however optional.

In subsequent steps 402-404, the changes between the reference image andthe main image are compared to detect and correct red eyes. The level ofred color changes considerably at red eye locations. In the referenceimage, the pupil is almost black, while the pupil is red in the mainimage. Information about the image changes can be used to determine redeye locations. In the detected areas, then the amount of redness (e.g.the red chrominance channel V of the YUV model, or the R component ofthe RGB image) needs to be reduced to remove the red eyes.

To this end, in a step 402, a red-eye detection is performed based onimage data of the main image. A red-eye detection algorithm usingtechniques such as for instance face detection or eye detection isdeployed to find red eyes in the main image. As an output, thisalgorithm produces information that characterizes red eye candidates inthe main image, such as for instance a binary map of potential red eyepixels, or a set of coordinates, shapes and sizes of potential red eyepixels. The algorithm may be parameterized so that the detection rate ishigh at the expense of high false positive rate. This is possible, sincethe following step 403 of flowchart 400, which is red-eye detectionrefinement, takes care of removing false positives. Allowing more falsepositives here may reduce the complexity of detection.

In a step 403, red-eye detection refinement is performed by excludingfalse positives from the information obtained from red-eye detection instep 402. Red-eye detection refinement is based on changes between thereference image and the main image, for instance by analyzing colorchanges in red eye candidate locations and around them. For instance,only red-eye candidate locations that are associated with a significantchange (e.g. from dark color in the reference image to red color in themain image) may be considered as true positives. It may also be possibleto utilize e.g. face detection information when the red-eye detectionrefinement is performed. If the face detection method is very robustthis information may also be utilized for limiting the number of red-eyecandidates in the red-eye detection of step 402.

Finally, in a step 404, red-eye correction is performed using a red-eyecorrection algorithm that utilizes the information on the red-eyecandidates obtained from step 403 and corrects the red-eye candidates inthe main image, thus obtaining a main image with removed red-eye effect.Therein, image data from the reference image may be used to correct thered-eye candidates. It is known to a person skilled in the art that aplurality of red-eye correction methods can be applied here. Examples ofsuch red-eye correction methods are described in the followingreferences:

-   -   Georg Petschnigg, Maneesh Agrawala, Hugues Hoppe, Richard        Szeliski, Michael Cohen, Kentaro Toyama. “Digital Photography        with Flash and No-Flash Image Pairs”. ACM Transactions on        Graphics (Proceedings of SIGGRAPH 2004), 2004.    -   GAUBATZ, M., AND ULICHNEY, R., 2002. “Automatic red-eye        detection and correction”. IEEE International Conference on        Image Processing, pp. 804-807.    -   PATTI, A., KONSTANTINIDES, K., TRETTER, D. AND LIN, Q., 1998.        “Automatic digital redeye reduction”. IEEE International        Conference on Image Processing, pp. 55-59.    -   Huitao Luo, Yen J., Tretter D. “An efficient automatic redeye        detection and correction algorithm”, ICPR 2004. Proceedings of        the 17th International Conference on Pattern Recognition, Volume        2, 23-26, Aug. 2004 Page(s):883-886.

A further example for a red-eye correction algorithm applicable in step404 will be discussed with reference to the flowchart 500 of FIG. 5below.

It should be noted that, instead of performing red-eye detection in step402 based on image data of the main image only, this red-eye detectionmay equally well be based on image data of both the reference image andthe main image. Furthermore, the red-eye detection refinement in step403 then may be performed based on image data of a single image, eitherthe reference image or the main image. This may reduce the computationaltime needed for the single-image red-eye detection refinement (e.g.based on face/eye shape detection), since fewer locations may need to besearched.

FIG. 5 presents a flowchart 500 of a further exemplary embodiment ofstep 205 of the flowchart of FIG. 2, i.e. a further exemplary way how todetermine image data of the main image with removed red-eye effect. Inthis exemplary embodiment, it is assumed that the image data of thereference image and the main image are represented by the Y, U and Vcomponents of the YUV color space model, wherein Y is the luminancecomponent, and wherein U and V are the chrominance components. Ofcourse, also other color spaces are feasible, such as the raw image datadomain or the RGB domain.

In a first step 501, a fuzzy likelihood map is determined, based on theY, U and V components of the reference image and the main image. Thismap provides a value in the range [0,1] for each pixel, wherein a largervalue describes that the pixel may likely be in a red-eye region.

In a step 502, the reference image and the main image are pixel-wisemultiplied by the fuzzy likelihood map, so that the potential red-eyepixels are emphasized in value, thus giving more weight in subsequentcomputations. This makes crisp thresholds unnecessary, which isadvantageous since threshold setting is generally a very difficult task,so that avoiding to set thresholds renders red-eye detection morerobust.

The following steps 503-507 are performed separately for at least thechrominance components U and V of the reference image and the mainimage, wherein the degree of the processing (for instance the amount ofmodification of the PCA transform matrix in step 505) may be differentfor the different components.

In step 503, a set of data is built from a chrominance component (U orV) of the weighted reference image and the weighted main image asobtained from step 502. This set of data may for instance comprise theentire V component of the weighted reference image and the entire Vcomponent of the weighted main image.

In step 504, a Principal Component Analysis (PCA) is computed for the(component-specific) set of data based on the covariance matrix of theset of data (e.g. weighted V component of main image and weighted Vcomponent of reference image). For instance, for the red chrominancecomponent V, a 2×2 PCA transform matrix v may be computed. The PCAtransform matrix provides the coefficients for applying the PCAtransform on the set of data, where the set of data is separated intotwo components: one expresses the greatest variability or energy of thejoint signal (the common part), and the other expresses the rest, i.e.the differences between the images (the different part). The actual PCAtransform of the set of data may not be performed.

In a step 505, the PCA transform matrix obtained in step 504 is modifiedin such a way that the smaller PCA component (the different part) issuppressed or reversed. For instance, the modification of the PCAtransform matrix may be a function of the ratio of the strongesteigenvalue to the weakest eigenvalue, or a fixed parameter. This yieldsa modified PCA transform matrix VV.

In a step 506, a transformation is determined based on the combinationof the PCA transform matrix obtained in step 504 and the modified PCAtransform matrix obtained in step 505. This transformation may forinstance be defined as the matrix product of the modified PCA transformmatrix vv and the transposed PCA transform matrix v, where thetransposed matrix applies the forward transform and the non-transposedmatrix an inverse transform. When applying the transposed transformationmatrix to the set of data, the set of data is thus transformed into anew coordinate system, and by applying the modified transformationmatrix, the set of data is transformed back from the new coordinatesystem, however in a modified way that suppresses or reverses thedifferent part and thus the differences between the image data of thereference image and the image data of the main image. The two transformsabove may not be applied separately, but as a combined matrix. Thecombined transformation then effectively performs forward PCA andmodified inverse PCA.

This embodiment, when performed with different parameters, can be usedfor reducing not only red eye artifacts, but also for adaptive fusing offlash and ambient lightning (re-lightning). This can be achieved byincorporating also the Y component into the PCA processing.

In a step 507, the transformation of step 506 is then applied to the setof data to obtain a chrominance component of a PCA-mixture image.Therein, the chrominance component obtained in step 507 represents amixture of the chrominance components of the weighted reference imageand of the weighted main image.

Steps 503-507 are performed separately for the U and V components of thereference image and the main image. Optionally, steps 503-507 may alsobe performed for the luminance component Y. If the Y component is notprocessed, the Y component of the main image may be used for theremaining steps of the flowchart 500.

In a step 508, the weighted reference image, the weighted main image andthe PCA-mixture image are combined under consideration of the fuzzylikelihood map to obtain a combined image. One such combination may bethe average of PCA-mixture image and the reference image weighted withthe fuzzy likelihood map combined with the main image weighted with thereverse fuzzy likelihood map. Then the potential red eyes have the pixelvalues mostly from the PCA-mixture and reference images, and pixels withsmall likelihood for red eyes have values mostly from the main image.

The procedure (in particular steps 503-508) may be repeated with varyingparameters for the respective color space components of the main andreference image (e.g. strong effect for R pair, smaller effect for G andB pairs; or strong effect for V pair, smaller effect for U and no effectfor Y pairs).

At this stage of flowchart 500, red pupils, for instance, have been madeless red.

In a step 509, the influence of the fuzzy likelihood map, which is alsopresent in the combined image of step 508, is divided out from thecombined image to obtain an adjusted combined image.

In a step 510, the intensity of those pixels where the color saturationof the processed image was decreased compared to main image by thepreceding steps of flowchart 500 is reduced in order to darken thepupils. These pixels may for instance be identified by determining thedifference between the adjusted combined image and the main image. Thisfinally yields a representation of the main image with removed red-eyeeffect.

As already stated above, it should be noted that the method according tothe flowchart 500 of FIG. 5 may also be used as a refinement and/orcorrection method (steps 403 and 404) in the flowchart 400 of FIG. 4. Inthis case, the analysis and correction may be made only for the areasthat are detected in step 402 and/or 403 of flowchart 400.

One optional feature in the above-described processing is re-lightningwhere also the luminance component Y is considered.

In a simple implementation, instead of factorization of the Y component,high values of the luminance component Y of the reference image areused, which preserves bright areas in the reference image that are notbright in the main image (e.g. lamps). Such an implementation may forinstance be a selection of the highest luminance values of independentlyscaled Y components of reference and main images. In a more advancedre-lightning implementation, the luminance component Y is included inthe PCA processing of steps 503-507.

The embodiment of the present invention described with reference to theflowchart 500 of FIG. 5 uses a reference image (captured with low levelof flash energy) and a main image (captured with a higher level of flashenergy) weighted by a fuzzy likelihood map to blindly suppress orreverse the locally weighted global color differences between the twoimages by means of a factor analysis method. This results in red-eyeremoval without thresholding or pattern recognition steps for findingeyes (eye-shapes) in the images. This makes the present embodimentrobust against missed eyes, which is a very annoying feature of somered-eye removal methods (the so-called husky dog effect).

It is readily clear for the skilled person that the embodiment of thepresent invention described with reference to the flowchart 500 of FIG.5 above can equally well be applied for artifact removal when thereference image is captured without flash. For instance, the viewfinderimage, which is generally captured without flash, may be used as areference image. Equally well, it may be determined that capturing thereference image with flash is not necessary, since the reference imagewithout flash is already bright enough.

Furthermore, the embodiment of the present invention as depicted in theflowchart 500 of FIG. 5 uses fuzzy weighting when computing potentialred-eye pixels, which makes the result spatially smooth, unlike othermethods where there may be visible edges between areas where red-eyeremoval has been applied or not applied.

According to the embodiment of the present invention as depicted in theflowchart 500 of FIG. 5, the price of the gained robustness againstmissed eyes and crisp color edges may be a slight tendency to alterother colored details than just red eyes. For instance, when comparingthe main image and the representation of the main image with removedred-eye effect, one may see some objects, especially red ones in shadow,to have somewhat different color. However, to the end user this is not aproblem as long as the image looks natural, because the user, whencapturing an image, will not see the main image, but only the(flash-less) viewfinder image. The small changes in colors are much moretolerable than missed eyes.

The embodiment of the present invention as depicted in the flowchart 500of FIG. 5 can be used in re-lightning by balancing also the intensity(luminance) differences in addition to the color differences (see steps503-507). The advantage for both red-eye removal and re-lightning isthat this embodiment computes the PCA transfer matrix based on thecurrent image data, and only the amount of modification of thisdata-driven matrix may be set by parameter values, for instance byparameter values that define how much the weakest PCA component isreduced or reversed. E.g. it may be set a strong effect for R pair,smaller effect for G and B pairs; or strong effect for V pair, smallereffect for U and medium effect for Y pairs). The parameters thus do notdefine by which amount the color components of the reference image andthe main image are combined. Furthermore, these amounts are computedon-the-fly, and the parameters do not specify the required rednessdifference threshold for the red-eye removal operation to take place.

The embodiment of the present invention as depicted in the flowchart 500of FIG. 5 is not limited to PCA; also other factor analysis methods maybe used, such as Independent Component Analysis (ICA), for separatingthe common part and the different part of the reference image and themain image. Also, this embodiment is not limited to using the YUV colorspace. Equally well, other color spaces can be used. For instance, theembodiment may be used in the RGB space either before or after Bayerde-mosaicing.

While there have been shown and described and pointed out fundamentalnovel features of the invention as applied to preferred embodimentsthereof, it will be understood that various omissions and substitutionsand changes in the form and details of the devices and methods describedmay be made by those skilled in the art without departing from thespirit of the invention. For example, it is expressly intended that allcombinations of those elements and/or method steps which performsubstantially the same function in substantially the same way to achievethe same results are within the scope of the invention. Moreover, itshould be recognized that structures and/or elements and/or method stepsshown and/or described in connection with any disclosed form orembodiment of the invention may be incorporated in any other disclosedor described or suggested form or embodiment as a general matter ofdesign choice.

1-37. (canceled)
 38. A method, comprising: receiving at least image dataof a reference image captured using a first level of flash energy andimage data of a main image captured using a second level of flash energythat is higher than said first level of flash energy; and determiningimage data of a third image at least based on said image data of saidreference image and said image data of said main image, wherein saidthird image is a representation of said main image with removedartifacts, wherein said determining of said image data of said thirdimage comprises performing a factor analysis of a set of data formedfrom said image data of said reference image and said image data of saidmain image, and applying a transformation obtained from said factoranalysis to said set of data to obtain image data of a fourth image, andwherein said image data of said third image is determined at least basedon said image data of said fourth image.
 39. The method according toclaim 38, wherein a temporal distance between the capture of thereference image and the main image is less than 100 ms.
 40. The methodaccording to claim 38, wherein said first level of flash energy is lessthan 10 percent of said second level of flash energy.
 41. The methodaccording to claim 38, wherein said reference image has at least one ofa lower quality, a lower sampling rate and a lower resolution than saidmain image.
 42. A computer-readable medium having a computer programstored thereon, the computer program comprising: instructions operableto cause a processor to receive at least image data of a reference imagecaptured using a first level of flash energy and image data of a mainimage captured using a second level of flash energy that is higher thansaid first level of flash energy; and instructions operable to cause aprocessor to determine image data of a third image at least based onsaid image data of said reference image and said image data of said mainimage, wherein said third image is a representation of said main imagewith removed artifacts, wherein said determining of said image data ofsaid third image comprises performing a factor analysis of a set of dataformed from said image data of said reference image and said image dataof said main image, and applying a transformation obtained from saidfactor analysis to said set of data to obtain image data of a fourthimage, and wherein said image data of said third image is determined atleast based on said image data of said fourth image.
 43. Thecomputer-readable medium according to claim 42, wherein a temporaldistance between the capture of said reference image and said main imageis less than 100 ms.
 44. An apparatus, comprising a processor, saidprocessor being configured to receive at least image data of a referenceimage captured using a first level of flash energy and image data of amain image captured using a second level of flash energy that is higherthan said first level of flash energy; and to determine image data of athird image at least based on said image data of said reference imageand said image data of said main image, wherein said third image is arepresentation of said main image with removed artifacts, wherein saiddetermining of said image data of said third image comprises performinga factor analysis of a set of data formed from said image data of saidreference image and said image data of said main image, and applying atransformation obtained from said factor analysis to said set of data toobtain image data of a fourth image, and wherein said image data of saidthird image is determined at least based on said image data of saidfourth image.
 45. The apparatus according to claim 44, furthercomprising a camera unit configured to capture said reference image andsaid main image.
 46. The apparatus according to claims 44, wherein atemporal distance between a capture of said reference image and saidmain image is less than 100 ms.
 47. The apparatus according to claim 44,wherein said first level of flash energy is less than 10 percent of saidsecond level of flash energy.
 48. The apparatus according to claim 44,wherein said reference image has at least one of a lower quality, alower sampling rate and a lower resolution than said main image.
 49. Theapparatus according to claim 44, wherein said factor analysis is aprincipal component analysis of said set of data, said principalcomponent analysis determining a common part and a different part withrespect to said image data of said reference image and said image dataof said main image.
 50. The apparatus according to claim 49, whereinsaid transformation is determined based on a transformation matrixobtained from said principal component analysis and a modifiedtransformation matrix determined to suppress or reverse said differentpart.
 51. The apparatus according to claim 44, wherein said processor isconfigured to determine a fuzzy likelihood map based on information fromsaid reference image and said main image, wherein said fuzzy likelihoodmap indicates whether parts of said main image contain an artifact ornot, and wherein said processor is configured to weight said image dataof said reference image and said image data of said main image with saidfuzzy likelihood map before said factor analysis is performed.
 52. Theapparatus according to claim 51, wherein said processor is furtherconfigured to determine said image data of said third image based onsaid image data of said reference image, said image data of said mainimage and said image data of said fourth image under consideration ofsaid fuzzy likelihood map.
 53. The apparatus according to claim 52,wherein said processor is further configured to remove an influence ofsaid fuzzy likelihood map from said image data of said third image. 54.The apparatus according to claim 44, wherein said processor is furtherconfigured to reduce an intensity of portions of said third image. 55.The apparatus according to claim 44, wherein said image data representscomponents of a color space model, wherein said processor is configuredto determine said image data of said third image for at least one ofsaid components of said color space model, and wherein said processor isconfigured to determine said image data of said third image for saidcomponents separately.
 56. The apparatus according to claim 55, whereinsaid color space model is the YUV model, and wherein said processor isconfigured to determine said image data of said third image for the Uand V component only.
 57. The apparatus according to claim 55, whereinsaid color space model is the YUV model, and wherein said processor isconfigured to determine said image data of said third image for the U, Vand Y component.